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PREFACE 


The  research  documented  in  this  technical  report  for  the  CAMS  at  the 
Flightline:  Nellis  Structured  Study  Test  Report  program  sponsored  by  the  Air 
Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Logistics 
Readiness  Branch  (AFRL/HESR),  Wright-Patterson  Air  Force  Base,  OH.  The 
University  of  Dayton  Research  Institute  performed  the  work  under  Delivery  Order 
#06  of  the  Technology  for  Readiness  and  Sustainment  (TRS)  contract  F33615- 
99-D-6001.  Maj.  Matthew  W.  Goddard  and  Capt.  Brian  Tidball  (AFRL/HESR)  are 
the  program  managers  for  the  effort. 
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1  Executive  Summary 


One  tool  the  US  Air  Force  uses  to  capture  all  aircraft  maintenance  actions  is  the  Core 
Automated  Maintenance  System  (CAMS),  a  Maintenance  Data  Collection  System  (MDC). 

When  the  term  “maintenance  documentation”  is  used  in  this  document,  it  refers  to  MDC  or 
CAMS  documentation,  not  to  be  confused  with  aircraft  forms  (781)  documentation,  a  related  but 
separate  issue.  CAMS  was  updated  across  the  Air  Force  in  2002  with  the  release  of  the  new 
CAMS  Graphical  User  Interface  (GUI);  thus,  the  term  “CAMS,”  as  used  in  this  document,  is 
assumed  to  refer  to  the  GUI  version  unless  otherwise  noted. 

Infrastructure  and  hardware  are  in  place  at  Nellis  AFB,  NV  for  CAMS  to  be  used  via  a 
mobile  device  on  the  flightline.  The  purpose  of  this  infrastructure  and  hardware  is  for  aircraft 
maintainers  to  document  aircraft  maintenance  actions  at  the  point  of  maintenance.  (The  term 
“point  of  maintenance”  refers  simply  to  the  location  of  the  maintainer  at  the  time  and  place 
aircraft  maintenance  is  completed.)  The  mobile  system  at  Nellis  AFB  is  referred  to  in  this  report 
as  the  Nellis  AFB  CAMS  at  the  Flightline  system,  or  simply  as  the  mobile  CAMS  system.  The 
Nellis  AFB  CAMS  at  the  Flightline  system  was  evaluated  by  an  AFRL  team  for  usability  per 
direction  from  HQ  USAF/ILMM.  The  AFRL  team  included  military  members  from  the  AFRL 
Human  Effectiveness  Directorate’s  Logistics  Readiness  Branch,  and  civilian  members  from  the 
University  of  Dayton  Research  Institute  (UDRI)  and  NCI  Corporation.  The  team  tested  and 
observed  eight  subjects  using  CAMS  at  the  Flightline. 

Objective  measurements  (i.e.,  time)  were  recorded  while  observing  maintainers 
documenting  simulated  aircraft  maintenance  using  both  stationary  CAMS  terminals  and  the 
mobile  system.  Results  indicate  that  when  the  technician’s  travel  time  to  and  from  the  aircraft  is 
considered  as  part  of  the  overall  task,  significant  time  savings  are  realized  by  use  of  the  mobile 
system;  however,  when  the  additional  time  required  to  sign-on  to  the  mobile  system  is  included 
in  the  analysis,  no  significant  time  savings  are  observed  for  the  mobile  system  as  compared  to 
the  stationary  method.  The  central  recommendation  from  these  objective  measurements  is  that 
the  mobile  system  sign-on  time  be  reduced  through  streamlining  of  the  sign-on  process  (e.g., 
combining  the  multiple  logins  required  in  the  mobile  condition)  or  through  more  use  of  a  more 
effective  wireless  network.  The  sign-on  time  is  the  key  identified  factor  precluding  significant 
time  savings  when  using  the  mobile  system. 

Subjective  data  was  also  collected  via  user  feedback  and  experimenter  observations. 

Two  key  subjective  findings  are  summarized  as  follows:  First,  the  sometimes  lengthy  response 
of  the  CAMS  system  (primarily  when  accessing  drop-down  menus)  lends  to  frustration  in  both 
the  stationary  and  mobile  conditions.  This  frustration  is  intensified  with  the  additional  delays  in 
accessing  these  menus  across  the  wireless  network  using  the  mobile  device.  Second,  participants 
preferred  the  stationary  CAMS  system  to  mobile  CAMS  because  of  the  additional  logon  required 
in  the  mobile  condition,  the  higher  perceived  reliability  of  the  stationary  system,  the  preference 
for  the  traditional  mouse  over  the  stylus,  and  simply  because  of  the  heat  stress  conditions  of  the 
Nellis  AFB  Flightline. 

In  consideration  of  both  the  objective  and  subjective  data,  the  recommendations  from  this 
study  are  that  both  the  mobile  (CAMS  at  the  Flightline)  and  stationary  CAMS  systems  be 
available  to  maintenance  technicians,  and  that  mobile  CAMS  sign-on  time  be  reduced. 
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2  Introduction 


This  document  describes  the  usability  test  conducted  by  the  Air  Force  Research 
Laboratory  and  the  University  of  Dayton  Research  Institute  to  assess  user  performance  with  the 
CAMS  at  the  Flightline  system  for  mobile  maintenance  data  collection.  This  exploration  of 
CAMS  at  the  Flightline  has  sprung  from  a  broader  Air  Force  initiative  to  apply  technology  to 
improve  the  maintainer’s  capabilities  at  the  point  of  maintenance.  Traditionally,  the  maintainer 
works  a  task  at  an  aircraft,  ventures  indoors  to  access  a  CAMS  terminal  and  database,  populates 
necessary  data  and  receives  feedback,  and  returns  to  the  flightline  to  complete  the  documentation 
at  the  aircraft.  The  advent  of  remote  devices  providing  access  to  the  CAMS  database  has 
permitted  maintenance  documentation  to  occur  entirely  while  the  technician  is  at  the  aircraft,  or 
at  the  point  of  maintenance.  This  test  consisted  of:  1)  observations  of  task  performance  in  two 
conditions  accessing  one  application  of  CAMS,  and  2)  feedback  questionnaires  about  the 
usability  of  both  the  baseline  and  test  devices.  The  two  performance  conditions  were  based  upon 
the  two  modes  of  operation  (i.e.,  stationary  and  mobile  use)  for  access  to  the  CAMS  database. 

The  stationary  CAMS  condition,  representative  of  the  traditional  maintenance  documentation 
process,  served  as  the  baseline  in  this  study.  The  mobile  CAMS  condition  reflects  one  possible 
application  of  the  point  of  maintenance  concept,  wherein  maintainers  accessed  the  same  GUI  as 
in  the  stationary  condition,  but  accessed  it  via  a  portable  laptop  connected  to  a  wireless  network. 

The  CAMS  GUI  is  the  newest  release  of  CAMS,  which  updates  the  previous  version  of 
CAMS,  the  CAMS  Green  Screen  interface.  Given  that  the  CAMS  GUI  had  only  recently  been 
released,  participants  in  this  study  were  also  asked  their  opinion  of  the  CAMS  GUI  in 
comparison  to  the  CAMS  Green  Screen  interface.  The  responses  are  included  in  the  results 
section. 

The  purpose  of  this  evaluation  was  to  assess  the  Nellis  AFB  CAMS  at  the  Flightline 
system  accessing  CAMS  via  wireless  frequency  against  the  desktop  device  accessing  CAMS  via 
wired  network  for  maintenance  documentation  (i.e.,  opening  work  orders,  listing  open  work 
orders,  and  closing  work  orders).  Both  of  these  methods  are  currently  in  use  at  Nellis,  the  site  of 
this  test.  The  CAMS  at  the  Flightline  device  used  was  the  Itronix  GoBook  MAX  with  the 
CAMS  GUI;  the  current  CAMS  system  consisted  of  a  standard  desktop  computer  using  the 
CAMS  GUI  interface.  Both  systems  were  connected  to  live  training  databases.  The  design 
allowed  for  U.S.  Air  Force  personnel  to  use  the  devices  in  a  flightline-type  setting  to  perform  the 
necessary  maintenance  documentation  tasks. 

The  conditions  tested  included  opening  work  orders,  listing  open  work  orders,  and 
closing  work  orders  in  both  the  stationary  CAMS  condition  and  the  mobile  CAMS  condition. 

This  study  focused  on  the  differences  and  similarities  between  the  stationary  CAMS  and  mobile 
CAMS  access  methods,  as  well  as  the  usability  of  each. 

2.1  Objectives 

This  usability  test  was  designed  to  address  the  following  usability  issues: 

•  For  the  MDC  tasks  of  opening  jobs,  listing  open  jobs  and  closing  jobs  on  the 
flightline,  how  does  mode  of  operation  (mobile  device  connected  with  CAMS,  or 
stationary  device  connected  with  CAMS)  affect  performance? 
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•  Which  mode  of  operation  do  users  prefer? 

•  Is  the  access  to  information  on  the  mobile  device  acceptable? 

2.2  Test  Methodology 

Usability  test  methods  applied  in  the  current  evaluation  are  based  on  principles  outlined 
by  Dumas  and  Redish  (1993).  In  accordance  with  these  usability  testing  methods,  design  of  the 
study  included  three  steps  (see  Figure  1).  First,  the  major  usability  issues  were  identified.  These 
included  concerns  about  the  general  use  (e.g.,  the  ability  of  the  device  to  operate  as  a  mobile 
access  device).  From  these  general  concerns,  specific  concerns  were  identified  (see  the 
objectives  above).  Finally,  methods  for  collecting  information  relevant  to  these  concerns  were 
identified.  Methods  included  multiple  metrics  for  each  concern  (e.g..  collecting  time 
measurements  for  completion  of  specific  tasks,  and  collecting  subjective  information). 


General 
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Metric 


Metric 


Metric 


Metric 


Metric- 

Metric 


Specific 
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Figure  1.  Usability  testing  methods  (Dumas  and  Redish,  1993). 


During  the  course  of  the  study,  time  data  measurements  w  ere  recorded  as  subjects 
completed  each  of  the  tasks  in  the  stationary  and  mobile  CAMS  conditions.  These  time 
measurements  permitted  an  objective  comparison  of  maintenance  documentation  tasks  in  these 
conditions. 
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In  addition  to  the  time  data  recorded  in  this  study,  participants  responded  to  questions 
pertaining  to  the  identified  usability  concerns.  For  this  data,  the  analysis  leverages  the  strength 
of  triangulating  the  information  gathered  for  each  specific  concern.  For  example,  the  specific 
concern  dealing  with  how  mode  of  operation  affects  performance  included  use  of  three  metrics: 
subject  ratings,  experimenter  observations  of  the  participant  while  using  the  device,  and  specific 
participant  comments  (both  written  and  verbalized,  during  the  test  and  in  post-test  discussions). 
Performance  times  further  validate  this  subjective  analysis.  Using  triangulation,  each  metric  is 
applied  to  confirm  findings  of  another  metric  so  that  if  all  three  metrics  identify  the  same  issue, 
the  strength  of  usability  concern  is  increased.  Similarly,  as  multiple  participants  identify  the 
same  usability  problem,  the  usability  problem  is  revealed.  Methods  used  to  analyze  data  are 
fully  addressed  in  the  Results  section  of  this  document. 

In  identifying  usability  concerns,  research  indicates  that  the  number  of  participants  need 
not  be  as  high  as  tests  aimed  at  determining  significance  (e.g..  Analysis  of  Variance  (ANOVAs), 
or  tests  of  correlation).  Virzi  (1992)  identifies  that  4  to  5  subjects  identify  80%  of  the  usability- 
problems  with  a  system,  and  that  additional  subjects  are  less  likely  to  identify  any  new  problems. 
The  current  test  included  8  subjects,  and  therefore  should  be  considered  to  be  of  relatively  high 
strength  in  identifying  usability  concerns  for  the  devices  tested.  When  considering  the  objective 
data,  8  subjects  alone  may  not  be  of  sufficient  power;  this  may  increase  the  likelihood  of  a  Type 
II  error  (i.e.,  not  finding  significance  when  there  is  actually  a  difference).  However,  when  the 
objective  data  is  viewed  with  the  subjective  feedback,  the  combined  data  should  provide  a  good 
overall  picture  of  the  usability  of  the  system. 
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3  Method 


3.1  Participants  and  Facility 

Eight  U.S.  Air  Force  personnel  at  Nellis  AFB  served  as  participants.  These  individuals 
were  assigned  to  the  57th  Aircraft  Maintenance  Squadron  (AMXS).  Participants  were  drawn 
from  the  A-10,  F-l  6  and  F-15  weapon  systems.  All  participants  were  either  crew  chiefs  or 
specialists.  Testing  was  conducted  at  the  57th  AMXS,  Nellis  AFB,  NV. 


3.2  Time  and  Schedule 

The  testing  was  conducted  from  19  May  through  22  May  2003.  Pre-testing  occurred  on 
the  first  day  of  testing.  Pre-testing  consisted  of  assuring  that  appropriate  software  was  loaded  on 
the  hardware  devices,  and  re-evaluating  all  scenarios  to  verify  that  the  same  types  of 
manipulations  were  required  across  devices.  Subjects  were  scheduled  on  a  non-interference 
basis;  that  is,  scheduling  was  arranged  based  solely  on  availability,  so  as  not  to  interfere  with 
subjects’  regular  work.  Each  subject  was  scheduled  for  2  hours;  no  one  required  more  than  this 
amount  of  time.  Following  the  pre-test  activity,  testing  occurred  on  the  remaining  three  days. 
Testing  on  all  three  days  lasted  for  most  of  the  day  (morning  and  afternoon). 

3.3  Test  Equipment  Requirements 

Hardware  used  in  the  test  included  the  items  listed  below. 

1 .  Itronix  GoBook  MAX  (750  MHz  Intel  Pentium  III  processor  with  256  KB  L2  cache 
512  MB  SDRAM  and  transflective  display)  ruggedized  laptop  with  CAMS  access 

2.  Extra  batteries  for  Itronix  GoBook  MAX 

3.  Battery  charger  for  Itronix  GoBook  MAX 

4.  Standard  CAMS  (1.8  GHz  Intel  Pentium  4  processor)  workstation  configured  with 
CAMS  access 

3.4  Data  Collection  Equipment 

Data  collection  instruments  included  the  following  items. 

1 .  One  (1)  video  tape  recorder  with  batteries 

2.  Blank  videocassette  tapes  (14) 

3.  Clipboards  (5) 

4.  Package  of  pens  (1) 

5.  Digital  camera  (1) 

3.5  Data  Collection  Packet 

Data  collection  packets  included  a  variety  of  forms.  Each  subject  completed  all  forms 
prior  to  the  end  of  the  test. 

1 .  Checklist 

2.  In-briefing 

3.  Consent  form 
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4.  Pre-test  questionnaires 

5.  Post-condition  questionnaires 

6.  Post-test  questionnaires 

3.6  Data  Collection  Team 

The  data  collection  team  consisted  of  three  individuals:  1)  one 
experimenter/video grapher,  2)  a  subject  matter  expert,  and  3)  a  study  coordinator.  The 
experimenter  conducted  the  in-briefing  and  out-briefing  sessions  and  administered  the 
questionnaires.  The  experimenter  also  operated  a  video  camera  in  order  to  record  participants’ 
actions  during  the  study.  The  subject  matter  expert  provided  the  majority  of  the  interaction  with 
test  participants.  The  study  coordinator  was  responsible  for  assuring  that  scheduling  was 
completed  for  all  participants,  and  that  all  hardware  and  software  were  available  for  the  test. 

3.7  Pre-Test  Requirements 

Several  activities  needed  to  be  completed  prior  to  beginning  this  test;  most  of  these 
activities  required  coordination  with  the  hosting  facility. 

1 .  Experimenters  required  time  to  view  the  CAMS  software.  This  software  was  loaded  on 
the  Itronix  GoBook  MAX  laptop  prior  to  this  test.  Experimenters  thoroughly  reviewed 
this  during  the  pre-test. 

2.  Experimenters  required  time  to  review  the  features  of  the  Itronix  GoBook  MAX  so  that 
they  would  be  familiar  with  the  device  prior  to  the  test.  This  was  accomplished  during 
the  pre-testing  day. 

3.  Experimenters  created  and  finalized  scenarios  for  the  subjects  to  use  during  the  test. 
Scenarios  were  developed  several  weeks  prior  to  the  test  and  finalized  on  the  pre-testing 
day. 

4.  Experimenters  were  granted  clearance  to  take  photos  at  Nellis  AFB.  Videotaped  data  and 
digital  photos  were  captured  during  the  test. 

5.  Experimenters  required  use  of  an  aircraft  parking  spot  on  the  flightline  for  testing.  The 
selected  parking  spot  was  as  close  as  possible  to  the  57th  AMXS  maintenance  dispatch 
location. 

3.8  Test  Procedure 

Test  participants  were  provided  an  in-briefing  in  a  conference  room  setting.  This  in¬ 
briefing  provided  an  overview  of  the  purpose  of  the  test.  At  this  time  they  also  completed  the 
consent  form,  and  completed  the  pre-test  questionnaire.  Each  participant  started  with  the 
baseline  stationary  CAMS  condition.  In  this  condition,  participants  used  a  CAMS  terminal  to 
complete  the  appropriate  documentation  for  an  aircraft  inspection.  First,  participants  initiated 
Windows  and  CAMS  logons  on  the  stationary  CAMS  terminal,  used  the  stationary  CAMS 
terminal  to  open  a  job  based  upon  appropriate  data,  and  then  logged  out  of  CAMS.  Participants 
then  logged  into  CAMS  again,  inspected  and  printed  an  electronic  list  of  all  open  jobs,  and 
logged  out  of  CAMS.  Finally,  participants  logged  into  CAMS,  closed  the  job  using  the 
stationary  CAMS  terminal,  and  logged  out  of  CAMS  a  final  time.  At  this  point,  the  individual 
logged  off  of  the  workstation.  Time  measurements  were  recorded  for  the  three  documentation 
tasks  participants  completed  in  the  baseline  condition.  After  the  participants  performed  the 
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documentation  (i.e.,  open  job,  list  open  jobs,  and  close  job),  they  were  asked  to  fill  out  a  post¬ 
condition  questionnaire. 

Following  the  completion  of  the  baseline  condition,  each  subject  used  the  laptop  with 
CAMS  in  the  mobile  condition  to  complete  the  same  documentation:  participants  opened  a  job, 
listed  open  jobs  and  closed  a  job  on  the  mobile  system.  Prior  to  beginning  the  mobile  condition, 
each  participant  received  training  on  the  mobile  device.  Once  trained  on  the  device,  individuals 
were  provided  the  option  of  initiating  Windows  and  network  logons  at  the  support  section  (where 
they  picked  up  the  device),  or  waiting  until  they  were  at  the  aircraft.  In  the  mobile  CAMS 
condition,  participants  again  were  required  to  logon  to  Windows,  as  they  had  in  the  stationary 
condition.  However,  the  mobile  CAMS  condition  required  an  additional  logon  step,  wherein 
participants  were  required  to  logon  to  the  wireless  network.  Once  logged  in  to  Windows  and  the 
network,  participants  could  logon  to  the  CAMS  database.  Either  before  or  after  this  logon, 
participants  walked  as  quickly  as  possible  to  the  aircraft  parking  spot  that  was  closest  to  the  57th 
AMXS  maintenance  dispatch  location.  At  the  aircraft  parking  spot,  participants  used  the  mobile 
CAMS  device  to  open  a  job  for  inspection,  and  then  logged  out  of  CAMS.  Participants  then 
logged  into  CAMS  again,  retrieved  and  reviewed  an  electronic  list  of  all  open  jobs,  completed 
the  steps  to  print  the  list  (but  did  not  actually  print  the  list),  and  logged  out  of  CAMS.  Finally, 
participants  logged  in  to  CAMS,  closed  the  job  in  the  CAMS  system,  and  logged  out  of  CAMS. 
Once  the  job  was  successfully  closed  participants  initiated  the  Windows  shut  down  and  network 
logoff,  and  walked  back  to  the  starting  area.  In  the  mobile  condition,  times  were  recorded  for 
each  of  the  three  tasks  completed  on  the  laptop.  Following  this  process  participants  were  asked 
to  fill  out  a  post-condition  questionnaire  and  a  post-test  questionnaire. 

The  CAMS  entries  completed  in  this  study  reflect  documentation  of  the  maintenance  that 
maintainers  complete  along  the  flightline.  The  test  plan  originally  included  an  order  parts  task, 
but  was  replaced  by  the  list/print  open  jobs  task.  This  was  a  creative  substitution  due  to  the  fact 
that  maintainers  at  Nellis  do  not  order  parts;  rather,  the  ordering  parts  task  is  a  supply 
responsibility.  It  is  commonplace,  however,  for  a  maintainer  to  retrieve  a  list  of  open  jobs 
affecting  an  aircraft,  and  print  out  that  list  for  use  on  the  flightline.  Thus,  the  list/print  open  jobs 
task  was  substituted  for  the  order  parts  task  in  this  test.  The  actual  printing  of  the  list  was 
simulated  due  to  the  fact  that  the  system  was  not  configured  with  printers  that  would  be  familiar 
to  the  participants.  Participants  completed  the  necessary  steps  on  the  laptop  to  arrive  at  the  print 
options  screen,  but  selected  the  Cancel  option  instead  of  the  Print  option  at  this  point. 

Travel  times  were  also  collected  for  inclusion  with  the  analysis  of  the  stationary  task.  For 
the  convenience  of  the  participants,  these  travel  times  were  collected  in  conjunction  with  the 
mobile  CAMS  condition;  that  is,  participants  were  timed  when  they  walked  from  the  57th  AMXS 
to  the  nearest  aircraft  parking  space  before  they  completed  the  mobile  task,  and  then  were  timed 
when  they  walked  from  that  parking  space  back  into  the  building  after  having  completed  the 
mobile  CAMS  condition.  The  average  of  these  times  was  then  determined  for  each  participant 
for  inclusion  in  specific  time  data  comparisons.  The  mobile  CAMS  allowed  for  real-time  remote 
data  interaction,  and  no  travel  time  was  required  to  input  and  receive  data  from  the  CAMS 
database;  individual  travel  times  were  recorded  to  better  document  the  activity  that  would  in  fact 
be  required  for  completion  of  the  stationary  CAMS  condition. 
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4  Results 


Analysis  of  the  data  collected  during  this  study  included  participants’  subjective 
evaluations  of  both  devices  across  a  series  of  characteristics,  as  well  as  a  direct  comparison  of 
actual  task  time  required  to  complete  specific  maintenance  documentation  activities.  These 
analyses  are  detailed  in  the  following  sections,  and  include  both  subjective  and  objective  results. 
It  is  important  to  note  that  a  triangulation  method  was  used  in  the  analysis  of  subjective  findings 
for  each  device.  Triangulation  was  implemented  as  defined  in  the  following  paragraphs. 

Ratings  for  both  devices  were  gathered  for  each  item  in  the  questionnaire  (e.g.,  for  each 
question),  and  mean  ratings  and  standard  deviations  were  then  calculated  on  each  item  for  that 
device.  Ratings  could  range  from  1  to  5  (one  was  positive  and  five  was  negative). 

A  priori,  criteria  were  set  that  1)  any  item  with  an  average  rating  of  >  3  indicated  a 
usability  problem,  and  2)  any  item  where  the  total  of  the  average  rating  plus  standard  deviation 
was  >  3  indicated  a  potential  usability  problem. 

Once  this  initial  analysis  was  complete,  user  ratings  were  plotted  on  a  cluster  graph  for 
items  that  indicated  usability  problems  or  potential  usability  problems.  User  comments  and 
observer  notes  were  then  analyzed  and  collapsed  to  assist  in  the  definition  and  clarification  of  the 
problem  or  potential  problem.  That  is,  user  comments  and  observer  notes  that  related 
specifically  to  the  item  were  added  to  the  analysis  to  assist  in  further  definition  of  the  problem  or 
potential  problem. 

Participants  completed  various  maintenance  documentation  tasks  (i.e.,  the  open  job,  list 
open  jobs,  and  close  job  documentation  tasks)  associated  with  aircraft  maintenance  in  two 
different  conditions:  1)  the  stationary  CAMS  condition,  which  represented  a  maintainer’s  typical 
documentation  experience,  and  2)  the  mobile  CAMS  condition,  which  utilized  a  wireless 
connected  laptop  device  to  allow  remote  completion  of  the  above  maintenance  documentation 
tasks.  Completion  times  for  these  tasks  and  subjective  feedback  are  discussed  in  the  analysis. 

Finally,  a  comparison  was  made  between  the  relevant  completion  times  for  the  various 
maintenance  documentation  activities.  Three  maintenance  documentation  tasks  were  completed 
in  both  the  stationary  and  mobile  CAMS  conditions.  Thus,  the  times  associated  with  opening  a 
job,  listing  all  open  jobs,  and  closing  a  job  were  compared  across  the  two  conditions. 


4.1  Subjective  Results 

Participants  completed  the  maintenance  documentation  activities  (i.e.,  open  job,  list  open 
jobs,  and  close  job)  on  the  stationary  CAMS  desktop  as  a  baseline.  Participants  entered  data 
typical  for  their  specialty  to  complete  the  documentation  tasks  successfully.  For  the  mobile 
CAMS  condition,  participants  completed  these  same  documentation  activities  at  the  aircraft 
using  a  laptop. 

After  completing  the  maintenance  documentation  tasks  in  each  condition,  participants 
rated  each  device  across  a  series  of  elements  using  a  5-point  scale,  where  one  was  positive  and 
five  was  negative.  Participants  rated  the  acceptability  of  each  device  for  such  tasks  as:  logging 
on  to  the  network,  logging  on  to  CAMS,  opening  a  job,  listing  open  jobs,  printing,  closing  a  job, 
using  the  drop-down  menus,  and  navigating.  Additionally,  participants  rated  the  acceptability  of 
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the  wireless  connectivity  of  the  mobile  CAMS  device,  and  comparatively  rated  their  frustration 
levels  when  using  each  system. 

In  all  categories,  the  average  ratings  in  both  the  stationary  CAMS  condition  and  the 
mobile  CAMS  condition  were  in  the  acceptable  range;  that  is  the  average  rating  in  each  category 
was  not  greater  than  or  equal  to  3.  Based  on  this  rating,  it  is  concluded  that  there  were  no 
demonstrated  usability  problems  in  either  condition. 

In  the  stationary  CAMS  condition,  the  average  rating  plus  one  standard  deviation 
remained  within  the  acceptable  range  for  all  categories  except  the  acceptability  of  the  drop-down 
menus.  This  indicates  that  the  only  potential  usability  problem  revealed  in  the  stationary  CAMS 
condition  for  this  study  related  to  the  acceptability  of  the  drop-down  menus  (Figure  2). 


Acceptable  Acceptable  Acceptable  Nor  Unacceptable  Unacceptable 
Unacceptable 

Figure  2.  Means  and  Deviations  -  acceptability  ratings  for  the  stationary  CAMS  condition. 

In  the  mobile  CAMS  condition,  the  calculation  of  the  average  rating  plus  standard 
deviation  revealed  four  potential  problems;  these  problems  were  related  to  the  acceptability  of 
the  network  logon,  printing  from  the  mobile  CAMS,  drop-down  menus,  and  wireless 
connectivity  (Figure  3). 


9 


Mobile  CAMS 


Figure  3.  Means  and  Deviations  -  acceptability  ratings  for  the  mobile  CAMS  condition. 

In  both  the  stationary  CAMS  and  mobile  CAMS  conditions,  individual  ratings  identified 
the  acceptability  of  drop-down  menus  as  a  potential  problem  ( figure  4 ).  Additionally,  in  the 
mobile  CAMS  condition,  acceptability  of  network  logon,  printing  and  wireless  connectivity  were 
defined  as  potential  problems.  Individual  ratings  for  these  three  usability  categories  are 
illustrated  in  Figures  5,  6,  &  7. 
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Acceptability  of  drop  down 
menus 
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Completely  Reasonably  Neither  Reasonably  Completely 

Acceptable  Acceptable  Acceptable  Unacceptable  Unacceptable 

Nor 

Unacceptable 

Figure  4.  Individual  ratings  for  acceptability  of  drop-down  menus  in  stationary  and  mobile 

CAMS  conditions. 

Individual  ratings  for  acceptability  of  drop-down  menus  in  both  the  stationary  and  mobile 
CAMS  conditions  indicated  potential  problems.  Seven  participants  provided  ratings  for  this 
item.  In  the  stationary  CAMS  condition,  5  participants  provided  ratings  in  the  acceptable  range: 

4  participants  rated  the  drop-down  menus  Completely  Acceptable,  and  1  participant  rated  the 
drop-down  menus  Reasonably  Acceptable.  However,  2  participants  provided  ratings  in  the 
unacceptable  range:  1  participant  rated  the  drop-down  menus  as  Neither  Acceptable  Nor 
Unacceptable,  and  1  participant  rated  the  drop-down  menus  as  Reasonably  Unacceptable. 

In  the  mobile  CAMS  condition,  4  participants  provided  ratings  in  the  acceptable  range  for 
this  item,  rating  the  drop-down  menus  Completely  Acceptable.  Three  participants  provided 
ratings  in  the  unacceptable  range:  1  participant  rated  drop-down  menus  in  the  mobile  CAMS 
condition  Neither  Acceptable  Nor  Unacceptable,  and  2  participants  rated  the  drop-down  menus 
Reasonably  Unacceptable. 

Participants  provided  some  general  comments  about  the  drop-down  menus:  3 
participants  pointed  out  that  the  drop-down  menus  were  useful  when  entering  such  items  as 
“aircraft  equipment  P/N’s  and  serial  #’s.”  Participants  also  provided  some  constructive  criticism 
that  included:  “I  like  the  idea  of  the  pop  down  windows  if  they  can  get  them  faster  and  more 
reliable”;  “the  only  problem  was  the  lagging  drop-down  menu.”  The  drop-down  menus  used  in 
the  CAMS  GUI  are  unlike  those  that  appear  in  standard  Windows  applications:  the  CAMS  GUI 
drop-downs  consist  of  windows  that  pop-up  (when  the  drop-down  is  selected)  and  are  populated 
with  the  relevant  data.  Participants  then  select  an  item  from  this  list,  which  populates  the 
associated  field,  and  then  close  the  pop-up  window. 


□ 


□ 


Potential  problem  with  use  of 
drop  down  menus 
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The  data  population  in  these  pop-up  windows  can  take  some  time.  When  an  individual 
selects  the  drop-down  list,  that  request  sends  a  query  to  the  CAMS  server  where  the  system 
searches  for  all  values  that  should  be  within  that  list,  and  then  returns  the  necessary  values  to 
populate  the  list.  As  this  list  can  contain  a  tremendous  number  of  items,  this  query-and-retum 
process  can  be  lengthy.  Furthermore,  added  time  associated  with  network  traffic,  as  in  the 
mobile  CAMS  condition,  could  further  exacerbate  the  wait.  Experimenters  noted  that,  in  both 
conditions,  participants  waited  approximately  45  to  60  seconds  for  some  drop-down  menus  to 
populate. 

Furthermore,  the  drop-down  menus  are  a  new  addition  to  CAMS.  Maintainers  are 
accustomed  to  referring  to  maintenance  books  to  determine  the  values  that  are  now  served  by  the 
drop-down  menus,  and  may  in  fact  recall  many  values  (e.g.,  the  necessary  work  unit  code)  from 
memory.  Thus,  while  the  current  method  for  populating  the  drop-down  menus  is  not 
expeditious,  these  menus  do  allow  maintainers  access  to  necessary  data  from  the  simplicity  of  a 
laptop.  Maintainers  have  the  option  to  cancel  the  request  of  any  drop-down  menu  that  responds 
too  slowly  (and  experimenters  noted  that  several  individuals  did  so  in  this  study).  At  this  point, 
they  can  either  type  in  the  required  information  from  memory  or  go  find  the  required  paper 
reference. 
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Potential  problem  with  logging  on 
to  network  in  the  mobile  condition 


Completely  Reasonably  Neither  Reasonably  Completely 

Acceptable  Acceptable  Acceptable  Unacceptable  Unacceptable 

Nor 

Unacceptable 

Figure  5.  Ratings  for  acceptability  of  network  logon  in  stationary  and  mobile  CAMS  conditions. 

Individual  ratings  for  acceptability  of  network  logon  indicated  potential  problems  in  the 
mobile  CAMS  condition.  Eight  participants  provided  ratings  for  this  item.  In  the  stationary 
CAMS  condition,  all  ratings  were  in  the  acceptable  range;  however,  in  the  mobile  CAMS 
condition,  six  ratings  were  in  the  acceptable  range  —  3  participants  provided  ratings  of 
Completely  Acceptable,  and  3  provided  ratings  of  Reasonably  Acceptable  -  but  2  participants 
provided  ratings  of  Reasonably  Unacceptable,  indicating  a  potential  problem  with  this  item. 
Participants  did  not  provide  comments  regarding  their  network  logon  ratings. 
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Experimenters  noted  that  several  participants  were  unfamiliar  with  the  network  logon 
process,  and  that  this  lack  of  familiarity  created  problems  for  the  users.  Not  only  was  there  an 
additional  logon  step  in  the  mobile  CAMS  condition  -  after  participants  signed  in  to  Windows, 
they  were  still  required  to  logon  to  the  network  before  logging  into  CAMS  -  but  this  logon  was 
different  from  the  Windows  logon,  and  was  often  not  easily  recalled  by  individuals  who  had  not 
recently  used  this  login  and  password.  Thus,  the  presence  of  an  additional  step  in  the  process 
was  likely  exacerbated  by  login  recall  difficulties.  Windows  and  network  logins  can,  in  fact,  be 
synched  for  user  convenience,  but  were  not  aligned  for  this  study.  In  the  stationary  CAMS 
condition,  the  act  of  logging  into  Windows  logged  the  user  onto  the  network,  as  well. 
Additionally,  the  system  was  already  on,  which  is  the  normal  state  of  these  computers.  These 
factors  reduced  any  wait  time  required  of  the  user  and  are  likely  reflected  in  the  more  positive 
perception  of  the  stationary  CAMS  condition  for  this  usability  item. 
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Figure  6.  Ratings  for  acceptability  of  printing  in  stationary  and  mobile  CAMS  conditions. 

Individual  ratings  for  the  acceptability  of  printing  indicated  potential  problems  in  the 
mobile  CAMS  condition.  Six  of  eight  participants  provided  ratings  for  this  item  in  the  mobile 
condition.  In  the  stationary  CAMS  condition,  all  ratings  were  in  the  acceptable  range;  however, 
in  the  mobile  CAMS  condition,  one  of  the  six  ratings  was  in  the  unacceptable  range,  where  1 
participant  rated  printing  in  the  mobile  condition  Reasonably  Unacceptable.  In  the  list  and  print 
open  jobs  task,  participants  in  the  mobile  CAMS  condition  only  completed  the  steps  necessary  to 
initiate  the  printing  task,  and  then  cancelled  out  of  the  print  window  without  actually  printing. 
Two  participants  chose  not  to  rate  the  printing  process  in  the  mobile  CAMS  condition  because 
they  did  not  actually  print  the  list  of  open  jobs. 

Experimenters  noted  that  the  simulated  print  task  created  some  confusion  for  the 
participants.  The  laptop  was  configured  according  to  the  requirements  of  the  support  section 
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where  the  hardware  was  checked  out;  that  is,  the  laptop  was  configured  with  printers  from  the 
57th  AMXS  maintenance  dispatch  location.  Thus,  when  participants  arrived  at  the  printer 
selection  window,  all  of  the  printers  were  unfamiliar.  This  may  have  suggested  that  the  printing 
process  with  the  mobile  device  would  be  unlike  printing  from  the  desktop.  Naturally,  if 
individuals  were  using  these  devices  on  a  regular  basis,  the  printer  options  would  be  defaulted 
based  upon  the  assigned  unit  of  the  individual.  In  addition,  maintainers  generally  print  lists  of 
open  jobs  from  the  stationary  desktop  CAMS  system  in  order  to  have  that  list  accessible  at  the 
aircraft.  As  this  list  of  open  jobs  is  available  on  the  mobile  device,  the  printed  list  may  become 
far  less  necessary. 
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Nor 
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Figure  7.  Ratings  for  acceptability  of  wireless  connectivity  in  mobile  CAMS  condition. 

Individual  ratings  of  acceptability  of  the  wireless  connectivity  in  the  mobile  CAMS 
condition  indicated  a  potential  problem.  Due  to  the  hard- wired  capabilities  of  the  stationary 
CAMS  condition,  wireless  connectivity  was  not  rated  in  that  condition.  In  the  mobile  condition, 
5  participants  provided  ratings  in  the  acceptable  range,  and  3  participants  rated  the  wireless 
connectivity  in  the  unacceptable  range:  2  participants  considered  the  wireless  connectivity 
Neither  Acceptable  Nor  Unacceptable,  and  1  rated  wireless  connectivity  Reasonably 
Unacceptable.  Participant  comments  indicated  that  there  continues  to  be  some  difficulty  with 
wireless  connectivity.  For  example,  one  participant  pointed  out  “if  you  move  into  a  dead  spot 
with  the  laptop  you  lose  everything  in  CAMS  and  have  to  go  to  a  live  spot  and  reboot.”  It  should 
be  noted  that  this  dead  spot  reported  by  the  participant  was  not  experienced  during  this  test; 
experimenters  attempted  to  reproduce  the  dead  spot  phenomena  based  upon  the  description  given 
by  the  participant,  but  were  unable  to  do  so. 
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Experimenters  noted  that  two  factors  were  likely  to  have  influenced  participant  responses 
regarding  wireless  connectivity.  First,  it  is  possible  that  the  additional  logon  required  for 
network  access  in  the  mobile  CAMS  condition  influenced  perception  of  wireless  connectivity,  as 
this  second  logon  was  only  necessary  to  connect  to  the  wireless  network.  Second,  experimenters 
noted  that  there  was  some  difficulty  with  the  logout  process:  when  participants  would  shut  down 
the  mobile  computer  at  the  end  of  the  test  session,  the  network  logoff  process  was  often  slow  or 
even  failed  entirely.  Such  a  problem  could  be  associated  with  both  network  logon  and  wireless 
connectivity,  and  may  well  be  reflected  in  the  potential  problems  participants  identified  in  both 
usability  assessments. 

After  completing  all  tasks  in  both  conditions,  participants  responded  to  questions 
regarding  their  preferences  for  access  methods  and  interfaces.  When  asked  which  of  the  two 
methods  individuals  preferred,  5  of  8  participants  indicated  a  preference  for  the  stationary 
CAMS,  2  preferred  the  mobile  CAMS,  and  1  individual  liked  both  equally  well  (Figure  8). 

Those  who  responded  that  they  preferred  the  stationary  CAMS  provided  reasons  such  as:  the 
logon  (and  logoff)  required  in  the  mobile  condition  (which  was  in  addition  to  the  Windows  and 
CAMS  logon  and  logoff  required  in  both  conditions),  the  perceived  higher  reliability  of  the 
desktop  connection,  a  preference  for  the  traditional  mouse  over  the  stylus,  difficulty  with  screen 
visibility  on  the  mobile  device,  and  a  preference  for  access  to  air  conditioning  during 
maintenance  documentation  tasks.  Both  of  the  individuals  w  ho  preferred  the  mobile  CAMS  did 
so  because  of  the  increased  efficiency  of  access  to  the  CAMS  information  while  at  the  flightline. 
Finally,  one  individual  pointed  out  “if  all  of  the  stationary  terminals  are  in  use  you  have  your 
own  personal  terminal.” 
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Figure  8.  Participants5  overall  terminal  preference. 
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Individuals  were  also  asked  the  degree  to  which  they  agreed  with  the  statement  “The 
mobile  CAMS  terminal  was  less  frustrating  to  use  than  the  stationary  CAMS  terminal.”  Three 
individuals  disagreed  with  this  statement,  and  5  participants  responded  that  they  neither  agreed 
nor  disagreed  with  this  statement  (Figure  9).  All  3  of  the  individuals  who  disagreed  with  this 
statement  commented  that  this  was,  at  least  in  part,  a  reflection  of  logon  difficulties  that  they 
experienced  in  the  mobile  CAMS  condition.  Participants  in  both  the  stationary  and  mobile 
CAMS  conditions  experienced  the  standard  CAMS  logon  difficulties;  however,  participants  in 
the  mobile  CAMS  condition  experienced  difficulty  with  the  network  logon  as  well. 


Frustration  Level 


Strongly  Agree  Neither  Disagree  Strongly 

Agree  Agree  Nor  Disagree 

Disagree 

Mobile  CAMS  system  is  less  frustrating  than  the  stationary  CAMS  system 

Figure  9.  Participants’  comparison  of  frustration  in  the  stationary  and  mobile  CAMS  conditions. 
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Finally,  individuals  indicated  their  preference  between  the  CAMS  GUI  and  the  CAMS 
Green  Screen  interface  (Figure  10).  Six  participants  indicated  that  they  preferred  the  CAMS 
GUI  (used  in  this  study  in  both  the  stationary  and  mobile  conditions),  and  two  participants 
preferred  the  CAMS  Green  Screen  interface.  Both  individuals  who  preferred  the  CAMS  Green 
Screen  interface  did  so  because  it  is  “much  faster”  than  the  GUI,  and  that  it  is  “not  as  susceptible 
to  Windows  based  crashes.”  As  a  Windows  interface  that  runs  on  a  Windows  based  computer, 
the  CAMS  GUI  is  affected  by  any  Windows  based  errors  or  crashes;  this  is  in  contrast  to  the 
CAMS  Green  Screens  (a  DOS-like  program),  perceived  by  participants  as  less  affected  by  any 
problems  within  the  Windows  system. 
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Figure  10.  Participants’  interface  preference. 

Those  who  preferred  the  CAMS  GUI  indicated  that  they  liked  the  drop-down  menus 
(particularly  if  the  drop-downs  can  be  made  “faster  and  more  reliable”),  and  that  they  liked  the 
mobility  associated  with  the  GUI  (in  that  any  Windows  based  network  connected  computer  can 
access  it).  Additionally,  one  participant  stated  that,  “once  you  leam  the  GUI  it  has  more  benefits 
and  uses.”  Whereas  the  CAMS  Green  Screen  interface  requires  that  the  user  recall  or  look  up 
necessary  information,  the  CAMS  GUI  provides  cues  in  the  form  of  drop-down  lists  to  assist  the 
user  with  appropriately  populating  the  information. 

These  subjective  findings  are  consistent  with  the  hypothesis  that  the  CAMS  GUI  is  an 
improvement  over  the  CAMS  Green  Screen  interface.  To  further  investigate  this  hypothesis,  a 
preliminary  empirical  analysis  was  performed  comparing  the  time  required  to  complete  the  open 
job  documentation  task  using  both  of  these  interfaces.  CAMS  GUI  conditions  were  represented 
in  the  analysis  of  the  stationary  and  mobile  conditions  for  opening  a  job  in  this  current  study. 
Data  for  the  CAMS  Green  Screen  condition  was  taken  from  data  collected  during  a  similar  study 
conducted  at  Hurlburt  Field,  the  POMX  Hurlburt  Initial  Structured  Study  (Gorman,  Donahoo, 
Quill,  Jemigan  &  Goddard).  This  preliminary  evaluation  indicates  a  potential  significant 
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difference  between  the  CAMS  GUI  and  the  CAMS  Green  Screen  interface.  The  POMX 
Hurlburt  Initial  Structured  Study  (Gorman  et  al.)  gathered  baseline  time  data  where  participants 
completed  the  open  job  documentation  task  using  the  CAMS  Green  Screen  interface;  this  was 
fundamentally  the  same  task  as  the  open  job  documentation  task  in  the  present  study  at  Nellis 
AFB.  Twelve  participants  took  part  in  the  Hurlburt  study;  data  was  gathered  for  eight 
participants  in  the  present  study  at  Nellis.  Due  to  unequal  n  in  these  studies,  an  independent 
samples  t-test  with  equal  variance  was  calculated  for  the  open  job  task  in  the  stationary  and 
mobile  CAMS  GUI  conditions  at  Nellis  AFB,  and  the  stationary  CAMS  Green  Screen  interface 
condition  at  Hurlburt  Field.  Figure  1 1  reflects  the  average  task  times  compared  in  this 
preliminary  evaluation. 


Stationary 


CAMS  Conditions 


Figure  11.  Average  time  of  open  job  documentation  tasks  across  CAMS  systems. 

Using  this  method,  the  open  job  documentation  task  using  the  mobile  CAMS  GUI  was 
significantly  faster  than  the  open  job  task  using  the  CAMS  Green  Screen  interface  on  a 
stationary  desktop  [t  (18)  =  2.529,  p  <  .05].  The  mobile  CAMS  GUI  condition  showed  an 
average  of  0:05:19  to  complete  the  open  job  task,  while  the  stationary  CAMS  Green  Screen 
interface  condition  showed  an  average  of  0:08: 13  to  complete  the  same  task. 

There  was  no  significance  when  comparing  the  open  job  documentation  task  in  the 
stationary  CAMS  GUI  condition  to  the  open  job  task  using  the  CAMS  Green  Screen  interface  on 
a  stationary  desktop  [t  (1 8)  =  1 .507,  p  >  .05].  The  stationary  CAMS  GUI  condition  showed  an 
average  of  0:06:13  to  complete  the  open  job  task,  while  the  stationary  CAMS  Green  Screen 
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interface  condition,  showed  an  average  of  0:08: 13  to  complete  the  same  task.  Analysis  of  these 
findings  indicates  that  there  might  be  a  real  advantage  to  using  the  mobile  CAMS  GUI  over  the 
stationary  CAMS  Green  Screen  interface.  Visual  inspection  of  the  means  indicates  that  there  is 
some  benefit  gained  from  the  use  of  the  CAMS  GUI;  however,  this  benefit  is  not  a  significant 
one,  as  is  illustrated  by  the  lack  of  statistical  significance  between  the  CAMS  Green  Screen  and 
the  stationary  CAMS  GUI  mean  task  times.  Yet  when  combined  with  mobile  computing,  the 
CAMS  GUI  does  show  potential  significant  improvement.  It  is  hypothesized  that  the  direct 
access  method  allowed  by  the  touch  screen  and  stylus,  when  combined  with  the  CAMS  GUI, 
promotes  enough  of  an  improvement  over  the  standard,  indirect  point-and-click  mouse  access 
method  to  account  for  the  significant  difference  between  the  mobile  CAMS  GUI  and  stationary 
CAMS  Green  Screen  interface  conditions. 

As  shown  in  the  results  of  the  current  study,  the  t-test  showed  no  significance  in  the 
difference  between  the  stationary  and  mobile  CAMS  GUI  conditions.  These  results  are 
discussed  in  the  Objective  Data  section  of  this  report. 

Interestingly,  this  preliminary  analysis  showed  a  compelling  loss  in  significance  when 
time  to  sign-on  to  the  network  was  incorporated  in  the  task  time  in  the  mobile  CAMS  GUI 
condition.  The  t-test  revealed  that  the  open  job  task  in  the  mobile  CAMS  GUI  condition  with 
network  sign-on  time  was  no  longer  significant  as  compared  to  the  open  job  task  using  the 
stationary  CAMS  Green  Screen  interface  [t  ( 1 8)  =  1 .41 1,  p  >  .05].  When  sign-on  time  was 
included,  the  mobile  CAMS  GUI  showed  an  average  time  of  0:10:02  to  complete  the  open  job 
task,  while  the  stationary  CAMS  Green  Screen  interface  showed  an  average  time  of  0:08:13  to 
complete  the  same  task  (see  Figure  12).  That  is,  the  advantage  gained  over  the  stationary  CAMS 
Green  Screen  interface  by  the  mobile  CAMS  GUI  is  lost  when  time  to  sign-on  to  the  network  is 
included  in  the  calculation. 


19 


10:48 


Hurlburt  CAMS  Green  Screen  -  Nellis  CAMS  GUI  -  Stationary  Nellis  CAMS  GUI  -  Mobile  With 
Stationary  Logon 

CAMS  Conditions  With  Mobile  Logon 


Figure  12.  Average  time  of  open  job  documentation  tasks  across  CAMS  systems  with  mobile 

logon. 

It  is  important  to  note  that  this  comparison  of  the  CAMS  GUI  and  CAMS  Green  Screen 
interface  is  a  preliminary  analysis.  Such  an  initial  evaluation  was  completed  for  discussion 
purposes,  and  does  not  have  enough  statistical  power  to  be  considered  valid  without  further 
testing.  For  validity  when  comparing  the  performance  times  for  task  completion  between  the 
CAMS  GUI  and  the  CAMS  Green  Screen  interface,  it  is  important  to  control  factors  such  as  the 
following:  subject  number  and  experience;  hardware  configuration;  locations;  task  requirements; 
network  considerations;  and  aircraft  being  maintained.  This  preliminary  CAMS  GUI  to  CAMS 
Green  Screen  interface  comparison  is  in  addition  to  the  primary  purpose  of  this  study  which  was 
to  compare  the  mobile  CAMS  to  the  stationary  CAMS.  While  limited  in  its  validity,  this 
preliminary  analysis  does  pose  some  intriguing  research  questions  which  should  be  considered 
for  future  research  efforts,  especially  with  regard  to  the  potential  need  for  improvements  to  logon 
procedures. 
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4.2  Objective  Results 

In  this  study,  participants  first  completed  various  maintenance  documentation  tasks  in  a 
baseline  condition,  a  condition  that  replicated  the  manner  in  which  the  maintenance 
documentation  is  usually  completed  (i.e.,  stationary  CAMS).  Then  participants  completed  the 
identical  documentation  tasks  (i.e.,  open  job,  list  open  jobs,  and  close  job)  in  the  mobile  CAMS 
condition.  Start  and  stop  times  were  recorded  for  each  of  these  tasks.  The  time  required  to 
complete  each  task  in  each  condition  was  then  compared  for  each  subject. 

Stationary  and  mobile  maintenance  documentation  task  times  were  evaluated  across  eight 
different  comparisons.  These  eight  comparisons  are  derived  from  analysis  that  balances 
exclusion  and  inclusion  of  1)  travel  time  and  2)  Windows  and  network  sign-on  times  for 
completion  of  the  maintenance  documentation  tasks  (i.e.,  open  job.  list  open  jobs,  and  close  job). 
Refer  to  Table  1  for  a  description  of  these  conditions. 


Table  1.  Definition  of  Comparisons  of  Task  Times 


Comparison  1 

Open  Job  Stationary  (with  travel)  vs.  Open  Job  Mobile 

Comparison  2 

Open  Job  Stationary  (without  travel)  vs.  Open  Job  Mobile 

Comparison  3 

Open  Job  Stationary  (with  travel  and  sign-on )  vs.  Open  Job  Mobile  (with 
sign-on) 

Comparison  4 

Open  Job  Stationary  (without  travel  and  with  sign-on)  vs.  Open  Job  Mobile 
(with  sign-on) 

Comparison  5 

List  Jobs  Stationary  (with  travel)  vs.  List  Jobs  Mobile 

Comparison  6 

List  Jobs  Stationary  (without  travel)  vs.  List  Jobs  Mobile 

Comparison  7 

Close  Job  Stationary  (with  travel)  vs.  Close  Job  Mobile 

Comparison  8 

Close  Job  Stationary  (without  travel)  vs.  Close  Job  Mobile 

Consideration  was  given  to  travel  time  required  when  accessing  the  stationary  CAMS 
terminal.  The  primary  difference  between  the  mobile  and  stationary  conditions  is  the  point  of 
access;  therefore,  travel  time  was  incorporated  into  each  discrete  comparison.  In  the  open  job 
and  list  open  jobs  tasks,  twice  the  average  travel  time  of  the  maintainor  w  as  added  to  the 
participant’s  overall  task  time  to  reflect  the  required  travel  from  the  aircraft  to  access  the  CAMS 
terminal,  and  to  reflect  the  required  travel  back  to  the  aircraft  to  continue  the  maintenance  or 
aircraft  781  forms  documentation.  In  the  close  job  task,  the  average  travel  time  was  added  in 
once  to  reflect  the  maintainer’s  return  from  the  aircraft  to  the  stationary  terminal,  where  the  job 
would  be  closed  in  CAMS. 

Conversely,  consideration  was  given  to  sign-on  time  required  for  the  mobile  device. 

Both  the  stationary  and  mobile  CAMS  conditions  required  that  the  users  sign-on  to  both 
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Windows  and  CAMS.  The  Windows  sign-on  time  recorded  in  the  stationary  and  mobile  CAMS 
conditions  reflected  the  entirety  of  the  time  it  took  for  the  user  to  activate  and  sign-on  to  the 
machine.  As  the  stationary  CAMS  terminals  were  left  on  at  all  times,  the  Windows  sign-on  time 
for  this  condition  simply  reflected  the  time  it  took  the  user  to  unlock  and  sign-on  to  the 
workstation.  On  the  other  hand,  participants  were  required  to  power  on  the  laptop  in  the  mobile 
CAMS  condition:  start  up  time  was  included  in  the  total  Windows  sign-on  time  required  in  this 
condition.  Furthermore,  while  the  Windows  sign-on  served  also  to  sign  the  user  on  to  the 
network  in  the  stationary  CAMS  condition,  the  mobile  CAMS  condition  required  that  the  user 
input  an  additional  logon  and  password  for  network  sign-on.  It  is  anticipated  that  maintainers 
would  sign-on  to  the  device  for  their  initial  access,  and  remain  signed  on  to  the  device  for  the 
duration  of  their  continuous  maintenance  activities.  For  this  reason,  both  Windows  and  network 
sign-on  times  have  been  included  only  with  the  initial  open  job  task. 

Finally,  while  it  was  assumed  that  maintainers  would  sign-on  to  the  mobile  device  for 
their  initial  access  and  remain  signed-on  for  the  duration  of  their  continuous  maintenance  , 
activities,  some  consideration  was  given  to  the  most  appropriate  representation  of  the  sign-on 
process  in  the  stationary  CAMS  condition.  Best  practices  assume  that  a  maintainer  will 
immediately  update  CAMS  anytime  the  status  of  an  aircraft  changes  (e.g.,  opening  a  job  to 
indicate  that  an  aircraft  is  broken,  or  closing  a  job  to  indicate  that  the  aircraft  maintenance  has 
been  completed);  however,  it  is  not  uncommon  for  technicians  to  access  the  CAMS  database 
only  a  few  times  through  their  shift,  and  update  all  information  at  that  time.  This  would  involve, 
then,  a  single  Windows  logon  for  several  maintenance  documentation  tasks  (e.g.,  open  jobs, 
close  jobs)  in  the  CAMS  database.  For  the  purposes  of  time  comparison,  this  more  efficient 
(although  not  more  effective)  approach  was  assumed.  Windows  logon  times  were  only  included 
in  specified  open  job  task  comparisons.  It  should  be  noted  that  the  list  open  jobs  task  would 
likely  be  completed  in  conjunction  with  another  task,  such  as  after  opening  or  closing  a  job,  and 
so  the  only  task  in  this  study  that  may  involve  additional  time  in  a  ‘best  practice’  scenario  would 
be  the  close  job  task. 

4.2.1  Comparison  1:  Open  Job  Stationary  (with  travel)  vs.  Open  Job  Mobile 

Comparison  1  contrasts  times  for  both  stationary  and  mobile  open  job  documentation 
tasks,  including  travel  time.  In  this  comparison,  it  is  assumed  that  the  maintainer  would  begin 
the  inspection  task  at  the  aircraft.  When  CAMS  access  is  required,  the  mobile  CAMS  access 
condition  would  allow  the  maintainer  to  remain  at  the  aircraft,  connecting  to  the  CAMS  database 
via  remote  frequency;  the  stationary  CAMS  desktop  condition  assumes  that  the  maintainer  would 
travel  to  the  CAMS  desktop  from  the  aircraft  to  input  information  about  the  job  and  receive  the 
appropriate  information  (i.e.,  Job  Control  Number,  or  JCN),  and  return  to  the  aircraft  to  complete 
the  inspection  task  and  aircraft  781  forms  documentation. 

Statistical  analysis  of  the  time  data  for  Comparison  1  shows  that  the  mobile  CAMS 
condition  yields  significant  improvements  over  the  stationary  CAMS  condition.  This  was 
explored  using  a  dependant  samples  t-test  conducted  on  the  total  task  time  for  the  open  job 
documentation  task  in  the  stationary  and  mobile  CAMS  conditions. 

To  include  travel  times  in  the  calculation,  experimenters  recorded  the  time  required  for 
each  maintainer  to  travel  from  the  CAMS  terminals  to  the  nearest  aircraft  parking  location.  This 
time  was  recorded  twice  for  every  participant,  and  these  times  were  averaged  to  determine  a 
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subject’s  travel  time  to  include  for  calculations  in  this  study.  To  portray  a  best  case  scenario, 
travel  times  to  the  nearest  aircraft  parking  location  were  used. 

As  hypothesized,  the  time  required  to  complete  the  documentation  task  in  the  mobile 
CAMS  condition  was  significantly  less  than  the  time  required  in  the  stationary  CAMS  condition 
when  travel  time  was  included  in  the  stationary  condition  [t  (7)  =  3.747,  p<.05].  The  mobile 
condition  showed  the  average  of  0:05:19  to  complete  the  open  job  documentation  task.  The 
stationary  condition  showed  an  average  of  0: 10:38  to  complete  the  open  job  task  when  travel  to 
and  from  the  aircraft  to  interact  with  the  stationary  system  was  included  in  the  calculation. 

Figure  13  shows  a  comparison  of  average  mobile  and  stationary  task  times  in  the  open  job 
comparison,  both  with  and  without  travel  times. 
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Figure  13.  Open  job  comparison  of  mobile  and  stationary  task  times  with  and  without  travel. 

4.2.2  Comparison  2:  Open  Job  Stationary  (without  travel)  vs.  Open  Job  Mobile 

Comparison  2  contrasts  task  times  for  both  stationary  and  mobile  open  job  documentation 
tasks;  this  comparison  does  not  include  travel  time  or  time  required  to  sign-on  to  the  system.  For 
this  comparison,  only  the  actual  times  to  complete  the  documentation  task  were  evaluated.  This 
excluded  the  time  required  for  the  maintainer  to  travel  from  the  aircraft  to  the  CAMS  terminal, 
and  the  time  required  to  return  to  the  aircraft  after  receiving  a  response  from  the  CAMS  system. 
Furthermore,  it  excluded  the  time  required  for  the  initial  wireless  network  sign-on  in  the  mobile 
CAMS  condition. 

A  dependent  samples  t-test  indicated  that  the  time  required  to  complete  the 
documentation  task  in  the  mobile  condition  did  not  significantly  differ  from  the  time  required  to 
complete  the  documentation  task  in  the  stationary  condition  [t  (7)  =  0.61 1 ,  p>.05].  In  the  mobile 
condition,  participants  spent  on  average  of  0:05:19  completing  the  open  job  documentation  task, 
while  in  the  stationary  condition  participants  spent  an  average  of  0:06:13  completing  the  open 
job  task.  These  results  are  shown  in  Figure  13.  This  figure  shows  that  time  savings  are  realized 
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through  saved  travel.  Statistically  significant  differences  in  time  required  for  performance  of  the 
documentation  task  itself  were  not  found. 

4.2.3  Comparison  3:  Open  Job  Stationary  (with  travel  and  siqn-on)  vs.  Open  Job 
Mobile  (with  siqn-on) 

Comparison  3  contrasts  task  times  for  both  stationary  and  mobile  open  job  documentation 
tasks,  including  travel  time  in  the  stationary  condition,  and  adding  to  both  conditions  the  time 
required  to  sign-on  to  Windows  and  the  wired  or  wireless  network.  Including  travel  time  with 
this  task  renders  the  most  realistic  representation  of  the  stationary  CAMS  condition;  as  well,  this 
comparison  considers  additional  time  the  maintainer  currently  requires  to  sign-on  to  each  mobile 
device.  This  sign-on  time  includes  a  single  logon  in  the  stationary  condition  that  signs  the  user 
into  Windows  as  well  as  the  wired  network;  in  the  mobile  condition,  this  sign-on  time  included 
two  logons  serving  to  sign  the  user  onto  Windows  and  the  wireless  network,  respectively.  In  this 
study,  Windows  and  network  start-up  times  were  recorded  with  the  open  job  documentation  task; 
during  actual  maintenance  activities,  it  is  likely  that  maintainers  will  initiate  the  start  up 
sequence  on  the  mobile  device  for  the  initial  use  and  will  leave  the  device  on  for  the  duration  of 
their  maintenance  inspection  and  documentation  tasks.  Note  that  Windows  and  network  sign-on 
times  were  incorporated  as  an  additional  variable  in  both  conditions  in  comparisons  3  and  4. 

Statistical  analysis  of  this  time  data  shows  that  there  was  no  significant  difference 
between  the  average  time  of  the  stationary  condition  and  the  average  time  of  the  mobile 
condition  in  the  open  job  documentation  task  where  both  travel  and  Windows  and  network  sign- 
on  were  included  in  the  relevant  conditions.  This  was  explored  using  a  dependant  samples  t-test 
conducted  on  the  total  task  time  (including  travel  and  Windows  and  network  sign-ons)  for  the 
open  job  documentation  task  in  the  stationary  and  mobile  CAMS  conditions. 

In  this  comparison,  the  time  required  to  complete  the  task  in  the  mobile  condition  did  not 
significantly  differ  from  the  time  required  to  complete  the  task  in  the  stationary  condition  [t  (6)  = 

1 .206,  p>.05].  The  mobile  condition  showed  the  average  time  of  0:10:02  to  complete  the  open 
job  documentation  task.  The  stationary  condition  showed  an  average  of  0: 1 1 : 1 1  to  complete  the 
open  job  task  when  travel  to  and  from  the  aircraft  to  interact  with  the  CAMS  system  was 
included  in  the  calculation.  Figure  14  shows  a  comparison  of  average  mobile  and  stationary  task 
times  for  the  open  job  task  where  the  time  for  Windows  and  network  sign-on  is  included  in  the 
stationary  and  mobile  CAMS  data. 
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Figure  14.  Open  job  comparison  of  mobile  and  stationary  task  times  with  and  without  travel 
where  Windows  and  network  sign-on  are  included. 

4.2.4  Comparison  4:  Open  Job  Stationary  (without  travel  and  with  siqn-on)  vs.  Open 
Job  Mobile  (with  siqn-on) 

Comparison  4  contrasts  task  times  for  both  stationary  and  mobile  open  job  documentation 
tasks  where  travel  time  is  not  included,  but  time  required  in  the  stationary  and  mobile  CAMS 
conditions  to  sign-on  to  Windows  and  the  wired  or  wireless  network  is  included.  This 
comparison  is  included  for  completeness. 

Statistical  analysis  of  the  time  data  in  this  comparison  shows  that  the  average  time  for  the 
stationary  condition  was  significantly  lower  than  the  average  time  for  the  mobile  condition.  This 
was  explored  using  a  dependant  samples  t-test  conducted  on  the  total  task  time  for  the  open  job 
documentation  task  in  the  stationary  and  mobile  CAMS  conditions.  The  time  required  to 
complete  the  task  in  the  mobile  condition  was  significantly  greater  than  the  time  required  in  the 
stationary  condition  when  sign-on  time  was  considered  in  both  conditions  and  travel  time  was 
not  incorporated  into  the  stationary  condition  [t  (6)  =  3.378,  p<.05].  The  mobile  condition  with 
sign-on  included  showed  the  average  of  0:10:02  to  complete  the  open  job  task.  The  stationary 
condition,  with  sign-on,  showed  an  average  of  0:06:48  to  complete  the  open  job  documentation 
task  when  travel  to  and  from  the  aircraft  to  interact  with  the  stationary  system  was  not  included 
in  the  calculation.  Figure  14  illustrates  the  average  mobile  and  stationary  task  times  for  this 
comparison. 
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4.2.5  Comparison  5:  List  Jobs  Stationary  (with  travel)  vs.  List  Jobs  Mobile 

Comparison  5  contrasts  task  times  for  both  stationary  and  mobile  listing  of  open  jobs 
where  travel  time  is  included  in  the  stationary  CAMS  condition.  This  travel  time  reflects  the 
time  the  maintainer  would  spend  traveling  from  the  aircraft  indoors  to  gain  access  to  the 
stationary  CAMS  system  to  list  the  open  jobs,  as  well  as  the  time  the  maintainer  would  spend 
returning  to  the  aircraft  with  that  list.  Again,  to  include  travel  times  in  the  calculation, 
experimenters  recorded  the  time  required  for  each  maintainer  to  travel  from  the  CAMS  terminals 
to  the  nearest  aircraft  parking  location.  This  time  was  recorded  twice  for  every  participant,  and 
these  times  were  averaged  to  determine  a  subject’s  travel  time  to  include  for  calculations  in  this 
study.  Twice  that  average  (to  indicate  travel  to  and  from  the  CAMS  terminal)  is  added  to  the  list 
open  jobs  task  time  in  the  stationary  CAMS  condition  for  this  comparison. 

As  hypothesized,  the  time  required  to  complete  the  task  in  the  mobile  condition  was 
significantly  less  than  the  time  required  in  the  stationary  condition  when  travel  time  was  included 
[t  (7)  =  10.821,  p<.05].  The  mobile  condition  showed  the  average  of  0:01 :50  to  complete  the  list 
open  jobs  task.  The  stationary  condition  showed  an  average  of  0:06:22  to  complete  the  list  open 
jobs  task  when  travel  to  and  from  the  aircraft  to  interact  with  the  stationary  system  was  included 
in  the  calculation.  Figure  1 5  shows  a  comparison  of  average  mobile  and  stationary  task  times  for 
this  comparison. 
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Figure  15.  List  open  jobs  comparison  of  mobile  and  stationary  task  times  with  and  without 

travel. 

4.2.6  Comparison  6:  List  Jobs  Stationary  (without  travel)  vs.  List  Jobs  Mobile 

Comparison  6  contrasts  task  time  for  both  stationary  and  mobile  listing  of  open  jobs 
where  travel  time  is  not  included  in  the  stationary  CAMS  condition.  That  is,  this  comparison 

reflects  time  that  the  user  was  actually  interacting  with  CAMS  over  the  connected  and  wireless 
networks,  and  does  not  consider  time  the  maintainer  would  have  spent  gaining  access  to  a  CAMS 
terminal. 
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As  hypothesized,  statistical  analysis  of  the  time  data  in  this  comparison  shows  that  there 
was  no  significant  difference  between  the  average  time  of  the  mobile  condition  and  the  average 
time  of  the  stationary  condition  for  the  listing  of  open  jobs  task  when  travel  was  not  included 
[t(7)  =  0.249,  p>.05].  This  was  explored  using  a  dependant  samples  t-test  conducted  on  the  total 
task  time  for  the  list  open  jobs  task  in  the  stationary  and  mobile  CAMS  conditions.  The  mobile 
condition  showed  the  average  of  0:01 :50  to  complete  the  list  open  jobs  task.  The  stationary 
condition  showed  an  average  of  0:01 :57  to  complete  the  list  open  jobs  task  when  travel  to  and 
from  the  aircraft  to  interact  with  the  CAMS  system  was  not  included  in  the  calculation.  Figure 
15  illustrates  average  mobile  and  stationary  CAMS  task  times,  both  with  and  without  travel. 
Again,  this  figure  shows  that  time  differences  between  platforms  are  due  to  travel. 

42.7  Comparison  7:  Close  Job  Stationary  (with  travel)  vs.  Close  Job  Mobile; 

Comparison  7  contrasts  task  times  for  both  stationary  and  mobile  close  job 
documentation  tasks,  and  includes  travel  time  required  to  close  the  maintenance  job  within 
CAMS  in  the  stationary  condition.  In  this  comparison,  the  participant’s  average  travel  time  was 
added  to  his  CAMS  access  time  only  once;  this  reflects  the  maintainer’s  action  of  returning  to  the 
stationary  CAMS  terminal  from  the  aircraft  to  close  the  job. 

Statistical  analysis  of  the  time  data  in  this  comparison  shows  that  the  mobile  condition 
yields  significant  time  improvements  over  the  stationary  condition  for  the  close  job  task  where 
travel  is  considered.  This  was  explored  using  a  dependant  samples  t-test  conducted  on  the  total 
task  time  for  the  close  job  task  in  the  stationary  and  mobile  CAMS  conditions.  The  time 
required  to  complete  the  task  in  the  mobile  condition  was  significantly  less  than  the  time 
required  in  the  stationary  condition  when  travel  was  included  [t  (7)  =  4.630,  p<.01].  The  mobile 
condition  showed  the  average  time  of  0:02:47  to  complete  the  close  job  task.  The  stationary 
condition  showed  an  average  of  0:05:3 1  to  complete  the  close  job  task  when  travel  from  the 
aircraft  to  interact  with  the  CAMS  system  was  included  in  the  calculation.  Figure  16  illustrates 
the  average  mobile  and  stationary  task  times  in  this  comparison. 
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Figure  16.  Close  job  comparison  of  mobile  and  stationary  task  times  with  and  without  travel. 

4.2.8  Comparison  8:  Close  Job  Stationary  (without  travel)  vs.  Close  Job  Mobile 

Comparison  8  contrasts  task  times  for  both  stationary  and  mobile  close  job 
documentation  tasks  where  travel  time  is  not  included  in  the  stationary  CAMS  condition.  That 
is,  this  condition  reflects  time  that  the  user  was  actually  interacting  with  the  CAMS  system  over 
the  connected  and  wireless  networks,  and  does  not  consider  time  the  maintainer  would  have 
spent  gaining  access  to  a  CAMS  terminal. 

Again  as  hypothesized,  the  time  required  to  complete  the  task  in  the  mobile  condition  did  not 
significantly  differ  from  the  time  required  to  complete  the  task  in  the  stationary  condition  [t  (7)  = 
.830,  p>.05].  The  mobile  condition  showed  the  average  of  0:02:47  to  complete  the  close  job 
task  The  stationary  condition  showed  an  average  of  0:03 : 1 6  to  complete  the  close  job  task  when 
travel  from  the  aircraft  to  interact  with  the  stationary  system  was  not  included  in  the  calculation. 
Figure  16  illustrates  average  mobile  and  stationary  task  times  in  this  comparison.  Again,  the 
addition  of  travel  time  is  the  significant  difference  in  the  close  job  documentation  tasks  on  the 
CAMS  platforms  tested. 
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5  Discussion 


The  purpose  of  this  study  was  to  provide  subjective  and  objective  data,  comparing  the 
usability  of  the  mobile  CAMS  device  to  the  current  stationary  CAMS  system  for  maintenance 
documentation  tasks. 

Overall  Time  Savings 

It  was  hypothesized  that,  due  to  elimination  of  travel,  the  mobile  CAMS  system  would 
allow  for  significant  improvements  in  the  time  required  to  complete  maintenance  at  the  aircraft, 
as  necessary  documentation  was  completed  at  the  point  of  maintenance.  For  example,  the 
current  method  for  completing  the  open  job  task  requires  that  the  maintainer,  upon  finding  a 
problem,  complete  paper  documentation  at  the  aircraft,  then  proceed  indoors  in  order  to  use  the 
CAMS  system.  The  maintainer  signs  in  to  the  CAMS  terminal,  opens  the  maintenance  job,  fills 
out  the  necessary  fields  within  the  system  and  sends  the  job,  and  then  waits  for  a  response  from 
the  CAMS  system  with  a  Job  Control  Number  (JCN).  At  this  point,  the  maintainer  signs  out  of 
the  CAMS  system,  returns  to  the  aircraft,  and  documents  the  JCN  in  the  aircraft’s  paper 
maintenance  forms.  Using  the  mobile  CAMS  system,  maintainers  can  access  information  in  the 
CAMS  database  using  a  laptop  connected  to  the  wireless  network,  and  can  complete  the  steps 
necessary  for  the  documentation  tasks  while  at  the  point  of  maintenance. 

It  should  be  noted  that  the  stationary  CAMS  condition  in  this  study  was  configured  to 
represent  the  best  possible  scenario  for  travel.  Time  data  was  collected  while  walking,  as 
quickly  as  possible,  to  and  from  the  nearest  aircraft  parking  location.  Given  that  convenient 
location  of  aircraft  is  fairly  infrequent,  it  is  reasonable  to  expect  that  travel  time  will  be  at  least 
equivalent  to  (and  probably  greater  than)  the  travel  time  included  in  this  study.  Also,  in  ordinary 
maintenance  activities  involving  the  stationary  CAMS  system,  the  maintainer  often  does  not 
enter  data  into  CAMS  until  the  end  of  his  or  her  shift.  Thus,  maintenance  data  in  CAMS  may 
not  actually  be  updated  for  up  to  eight  hours.  In  such  a  case,  the  maintenance  information  at  the 
aircraft  will  not  be  current  until  after  the  maintainer  has  received  responses  from  the  CAMS 
system  and  updated  the  documentation  at  the  aircraft.  On  the  other  hand,  with  the  mobile  CAMS 
system  the  database  information  and  documentation  at  the  aircraft  are  more  likely  to  be  current, 
as  maintainers  can  access  CAMS  to  update  it  as  soon  as  maintenance  is  completed. 

As  hypothesized,  this  study  did  show  significant  timesavings  in  the  mobile  CAMS 
condition  as  compared  with  the  stationary  CAMS  condition  when  travel  was  considered 
(Comparisons  1,  5,  &  7). 

Knowledge  transfer 

When  travel  was  not  included  in  the  time  measurements,  there  were  no  significant 
differences  between  the  open  job,  list  open  jobs,  and  close  job  documentation  tasks 
(Comparisons  2, 6,  &  8).  This  would  suggest  that  the  mobile  CAMS  system  is  comparable  to  the 
stationary  CAMS  system  in  these  tasks,  as  would  be  expected  since  the  interfaces  are  the  same. 
The  lack  of  difference  found  between  these  conditions  is  important.  Users  do  not  have  to  learn 
two  different  interfaces  (one  for  the  mobile  and  one  for  the  stationary  CAMS  access);  thus, 
learning  to  use  one  translates  to  equivalent  performance  on  both. 


Logon  issues 

To  account  for  logon  activities,  this  study  included  Windows  sign-on  time  in  the  open  job 
task  in  both  the  stationary  and  mobile  CAMS  conditions.  This  assumes  that  the  mobile  device 
will  be  left  on  for  the  duration  of  the  maintenance  activities,  and  that  the  maintainer  will  logon  to 
the  stationary  CAMS  system  only  one  time  to  complete  the  open  and  close  job  tasks.  While  time 
to  complete  the  close  job  task  did  not  significantly  differ  between  the  stationary  and  mobile 
CAMS  conditions,  this  calculation  was  performed  assuming  that  the  maintainer  did  not  have  to 
login  to  Windows  before  completing  the  close  job  documentation  task. 

It  is  interesting  that  when  logon  and  travel  time  were  included,  there  were  no  significant 
differences  in  overall  documentation  task  times.  However,  when  the  open  job  task  comparison 
did  not  include  travel  time  but  included  the  time  required  to  sign-on  to  the  network,  the  mobile 
condition  took  significantly  longer.  This  difference  due  to  logon  time  is  validated  by  the 
subjective  feedback  provided  by  participants  that  indicated  that  network  logon  and  wireless 
connectivity  were  potential  problems. 

There  are  a  number  of  factors  that  could  have  contributed  to  the  less  favorable 
performance  (and  thus,  user  perception)  of  the  network  logon  for  the  mobile  device.  First,  the 
stationary  CAMS  condition  consisted  of  a  Windows  logon  only,  whereas  the  mobile  condition 
required  both  a  Windows  and  a  network  logon.  Not  only  did  this  add  an  additional  step  to  the 
process,  but  the  network  logon  in  the  mobile  CAMS  condition  required  an  additional  password 
that  was  often  not  easily  recalled  by  users.  Some  maintainers  simply  did  not  recall  their 
password,  either  because  they  did  not  login  to  the  wireless  network  on  a  regular  basis,  or  because 
they  had  not  logged  into  the  wireless  network  for  such  a  length  of  time  that  they  required  new 
passwords.  The  CAMS  logon,  on  the  other  hand,  was  rated  by  individuals  as  acceptable  across 
both  conditions.  These  findings  indicate  that  difficulties  with  the  logon  process  need  to  be 
addressed.  A  synchronized  logon  for  Windows  and  the  wireless  network  should  be  required. 

Wireless  network  delays 

In  addition  to  the  objective  results,  subjective  results  suggest  that  interacting  with  the 
CAMS  database  via  the  wireless  network  may  have  exacerbated  some  issues.  The  additional 
network  logon,  discussed  above,  was  defined  as  a  potential  problem.  The  drop-down  menus  — 
that  were  a  potential  problem  in  both  conditions  -  were  more  slowly  populated  in  the  mobile 
condition;  in  both  conditions,  the  drop-down  menus  took  as  long  as  60  seconds  to  populate. 

Some  participants  would  simply  cancel  out  of  the  drop-down  menu  and  enter  a  code  from 
memory  rather  than  wait  for  the  system  to  return  the  necessary  data.  (Of  course,  this  strategy  is 
only  effective  if  the  individual  recalled  the  number  or  code  that  they  were  looking  for  in  CAMS.) 
Experimenters  did  note  that,  although  participants  experienced  lengthy  waits  while  the  drop¬ 
down  menus  populated  in  both  conditions,  those  in  the  mobile  condition  often  experienced 
longer  and  more  variable  delays,  most  likely  due  to  the  variance  of  traffic  across  the  wireless 
network. 

Another  finding  from  the  subjective  evaluation  revealed  that  maintainers  frequently 
experienced  a  long  logoff  time  in  the  mobile  condition  (at  times,  this  logoff  process  simply  did 
not  complete).  In  some  cases,  it  was  unclear  whether  the  individual  was  logged  out  or  not.  This 
was  due  in  part  to  the  process  required  for  the  system  to  recognize  a  logoff  request,  and  was 
aggravated  by  the  lag  time  of  the  wireless  connection.  Overall,  the  logoff  process  created 
difficulties  for  other  individuals  upon  logging  into  the  system,  and  thus  perpetuated  any 


frustration  experienced  with  the  additional  logon  and  ambiguous  logoff  required  in  the  mobile 
condition. 

Printing 

Interaction  with  the  CAMS  database  over  a  wireless  network  promotes  the  potential  for 
confusion  about  printing  processes  and  locations.  User  comments  regarding  the  potential 
problem  of  printing  in  the  mobile  condition  related  primarily  to  the  need  for  coordination  and 
set-up  to  ensure  proper  printer  settings  are  available  for  each  user. 

User  perceptions 

Based  on  other  user  feedback  and  experimenter  observations,  it  is  likely  that  the 
sometimes  lengthy  response  time  of  the  CAMS  system  -  in  the  drop-down  menus,  and  elsewhere 
-  lends  to  frustration  with  the  system,  especially  in  the  mobile  condition.  Five  of  eight 
participants  found  the  two  systems  to  be  equally  frustrating,  indicating  some  ambivalence  toward 
both  systems.  However,  the  remaining  three  participants  determined  that  the  mobile  CAMS 
system  was  more  frustrating.  Given  the  additional  difficulties  incurred  with  gaining  and 
terminating  access  to  the  wireless  network,  this  response  is  not  unexpected. 

Similarly,  the  majority  of  participants  preferred  the  stationary  CAMS  system  to  the 
mobile  CAMS  system.  The  reasons  provided  by  participants  include,  most  notably,  the 
additional  logon  process  in  the  mobile  condition,  as  well  as  the  higher  perceived  reliability  of  the 
stationary  terminals,  the  preference  for  the  traditional  mouse  over  the  stylus,  and  the  need 
(somewhat  unique  to  Nellis  AFB)  to  get  inside  periodically  and  away  from  the  heat  of  the 
flightline.  One  final  preference  comparison  was  made  between  the  current  CAMS  GUI  and  the 
recently  replaced  CAMS  Green  Screen  interface.  The  majority  of  participants  preferred  the 
CAMS  GUI,  which  “has  more  benefits  and  uses.”  It  should  be  noted  that,  while  maintainers  may 
prefer  the  CAMS  GUI  over  the  CAMS  Green  Screen  interface,  any  timesavings  offered  by  the 
CAMS  GUI  (specifically,  the  mobile  CAMS  GUI)  may  be  mitigated  by  unresolved  difficulties 
with  the  wireless  network  sign-on  process. 

Conclusions 

In  conclusion,  the  mobile  CAMS  condition  was  faster  than  the  stationary  CAMS 
condition  when  necessary  travel  time  was  included.  However,  the  mobile  unit  had  an  additional 
network  logon  that  hindered  the  participants  from  completing  the  open  job  documentation  task  as 
quickly  as  anticipated,  as  well  as  a  longer  lag  time  due  to  the  wireless  connectivity.  Subjective 
data  revealed  that  the  drop-down  menus  were  potential  problems  in  both  the  stationary  and 
mobile  CAMS  conditions. 

The  results  of  this  study  indicate  that  there  are  benefits  to  having  both  the  mobile  and 
stationary  systems  in  use.  The  mobile  CAMS  system  allows  the  aircraft  maintenance  job 
documentation  data  to  be  updated  in  the  timeliest  fashion,  at  the  point  of  maintenance.  This 
eliminates  travel  time,  and  increases  the  accuracy  and  currency  of  the  data  in  the  CAMS 
database.  In  addition,  with  the  mobile  CAMS  system,  maintainers  can  reference  the  data  they 
need  while  at  the  aircraft,  which  may  greatly  reduce  the  need  for  tasks  similar  to  the  print  open 
jobs  task.  As  one  maintainer  suggested,  the  availability  of  both  systems  resolves  the  ever-present 
concern  of  system  access,  noting  that,  at  the  very  least,  “if  all  the  stationary  terminals  are  in  use 
you  have  your  own  personal  terminal.”  On  the  other  hand,  maintainers  are  currently  more 
familiar  and  comfortable  with  the  stationary  system,  and  this  system  also  allows  relief  from 
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oppressive  or  inclement  weather.  The  recently  implemented  CAMS  GUI  is  already  an 
improvement  over  the  CAMS  Green  Screen  interface;  however,  results  indicate  that  potential 
problems  exist  with  the  drop-down  menus.  In  their  current  forms,  the  use  of  both  systems  may 
be  the  most  beneficial  option,  offering  the  convenience  and  efficiency  of  mobile  access  as  well 
as  the  stability  and  familiarity  of  the  stationary  system. 
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6  Recommendations 


The  results  of  this  study  indicate  that  there  are  considerable  gains  to  be  made  in  the 
efficiency  of  maintenance  documentation  tasks  and,  thus,  maintenance  tasks  overall.  Potential 
gains  include  time  savings  in  data  entry  and  retrieval,  error  reduction  due  to  reduced  memory 
load,  and  more  timely  entry  of  data  into  CAMS. 

For  this  study,  the  reported  time  savings  where  necessary  travel  time  was  incorporated 
into  task  time  were  significant.  Minutes  saved  each  time  a  job  is  opened  or  closed  enable  a  more 
efficient  and  effective  maintainer.  Furthermore,  average  travel  time  was  determined  based  on 
the  location  of  the  nearest  aircraft  parking  location.  As  travel  time  will  vary  widely  based  upon 
the  flightline  size  of  various  installations,  it  should  be  noted  that  Ilightline  size  should  clearly  be 
considered  when  estimating  the  potential  time  savings  a  unit  might  gain  by  incorporating  mobile 
devices  that  might  be  utilized  to  interact  with  MDC  systems  from  the  point  of  maintenance. 

The  following  recommendations  are  a  result  of  this  study: 

•  Allow  both  stationary  and  mobile  CAMS  access  devices  to  be  put  to  use.  While  the 
stationary  system  has  proven  its  viability  and  provides  the  technician  with  relief  from 
unfavorable  weather  conditions,  the  mobile  device  provides  more  access  to  information 
while  at  the  aircraft,  and  significantly  improves  the  timeliness  of  data. 

•  Improve  the  speed  and  usability  of  drop-down  menus.  The  intent  of  the  drop-down 
menus  is  to  provide,  within  the  GUI,  any  and  all  relevant  eodes  or  characters  that  may  be 
used  to  populate  a  field.  This  should,  then,  allow  the  maintainer  to  navigate  through  the 
CAMS  GUI  without  requiring  additional  reference  manuals.  1  lowever,  the  number  of 
values  returned  is  often  so  great  that  population  of  these  windows  takes  an  inordinate 
amount  of  time;  some  maintainers  simply  choose  not  to  use  this  resource  because  of  this 
(and  are  forced  to  recall  the  information  from  memory  or  find  the  book  to  look  it  up),  and 
others  endure  the  often  frustrating  wait.  The  process  of  populating  the  drop-down  menus 
should  be  reassessed  to  determine  if  the  values  could  be  accessed  w  ithout  such  an 
extensive,  time-consuming  database  search.  In  addition,  the  current  drop-down  menus  do 
not  function  in  a  manner  consistent  with  the  expectations  of  a  standard  Windows  user. 
When  an  individual  selects  a  drop-down,  he  or  she  is  not  given  a  scrollable  list  that 
populates  the  field  once  a  value  is  clicked  upon.  Instead,  selection  of  a  drop-down  menu 
causes  an  additional  window  to  open  up.  Users  must  search  through  this  window  for  the 
desired  value,  choose  it,  and  close  the  window.  This  process  is  more  involved,  and  thus 
less  acceptable,  than  the  traditional  Windows  drop-down  menu  format. 

•  Improve  the  mobile  logon  process.  In  the  stationary  CAMS  condition,  users  were 
required  to  input  only  one  logon;  this  is  because  Window  s  and  network  logons  were 
coordinated  such  that  both  processes  were  achieved  w  ith  a  single  logon.  This 
coordination  was  not  in  place  for  the  mobile  CAMS  condition,  and  user  feedback 
indicated  that  the  frustration  level  was  increased  because  of  the  complications  with  this 
additional  step.  The  system  should  automatically  provide  a  coordinated  logon  for  each 
user. 
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•  Address  difficulties  with  the  wireless  network.  Not  only  is  there  a  perception  that  the 
wireless  network  is  not  as  reliable  (i.e.,  it  has  “dead  spots”),  but  traffic  over  the  wireless 
network  is  highly  variable  and  can  create  more  slow  and  frustrating  interaction  with  the 
mobile  CAMS  system.  It  is  likely  that  some  of  this  frustration  will  be  alleviated  as 
technology  improves. 

•  Continue  to  look  for  technology  improvements.  Continued  consideration  of  the  use  of 
the  CAMS  GUI  across  multiple  styles  of  mobile  computing  devices  is  recommended. 
While  the  mobile  computer  used  in  this  study  was  state-of-the-art  at  the  time,  technology 
continues  to  evolve.  In  the  current  study,  there  was  some  concern  that  the  device  was  not 
entirely  sunlight  readable,  and  some  participants  indicated  a  preference  for  use  of  the 
mouse  over  the  touch  screen  and  stylus  of  the  laptop.  Revels  et  al.  explored  maintainers 5 
reactions  to  various  suites  of  electronic  devices,  including  a  variety  of  access  methods. 
One  result  of  this  study  indicated  that  maintainers  preferred  use  of  a  track  ball  mouse,  or 
a  stylus  attached  to  the  electronic  device  by  a  retractable  tether.  Future  studies  may 
explore  the  most  appropriate  input  devices  for  maintainers  entering  data  at  the  point  of 
maintenance. 

•  Configure  printer  for  the  user’s  location.  Ensure  that  printers  are  pre-configured  for 
users  so  that  they  can  easily  select  the  appropriate  printer  for  their  location.  This  should 
not  be  accomplished  as  the  user  checks  out  the  system,  but  should  be  an  administrative 
function  accomplished  before  users  arrive  to  check  out  a  device. 

•  Rework  the  log-out  procedure  for  the  mobile  device.  Allow  the  user  to  send  a  simple 
command,  and  then  have  the  network  (not  the  wireless  network)  log  them  out.  Ease  in 
accomplishing  this  task  will  reduce  both  the  errors  and  frustration  with  the  mobile  device. 

•  Keep  the  interface  consistent  between  the  mobile  and  stationary  platforms.  The 

consistency  of  interface  used  will  greatly  affect  how  and  when  technicians  use  the  device. 
If  the  same  interface  is  available  on  each  device,  maintainers  will  be  more  likely  to  enter 
data  at  the  point  of  maintenance.  Ultimately,  this  will  improve  the  timeliness  of  data. 

•  Explore  the  improvement  of  the  CAMS  GUI  over  the  CAMS  Green  Screen  interface 
within  a  pool  of  unseasoned  users.  Such  a  pool  of  users  will  more  effectively  reveal 
any  advantages  gained  by  the  additional  information  provided  within  the  CAMS  GUI, 
and  will  likely  offer  the  most  unbiased  comparison  of  these  two  methods  of  CAMS 
access. 

This  study  represents  an  important  first  step  in  assessing  the  feasibility  and  potential 
benefits  of  the  mobile  CAMS  system.  Further  refinement  of  this  technology  may  realize  more 
definitive  improvements.  Overall,  findings  from  this  study  are  promising  and  suggest  important 
questions  for  future  research. 
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